Network Management by Delegation Early Draft: Do Not Distribute

ion of the programming interface in which to encode management scripts. We will do that by providing a very high-level special-purpose management scripting language with special primitives to simplify the encoding of such an application. From an implementation perspective, these new primitives can be considered macroextensions that can be compiled into any general-purpose programming language (e.g. C), or they can be interpreted by the agent. 7 Models for Manager/Agent Interaction In this section we review existing models of process interactions, and evaluate their suitability for resolving the micro-management problem. The client server interaction paradigm is widely used for structuring distributed software. For example, in the remote procedure call (RPC) model [9], a server exports a number of xed procedures that can be invoked remotely by clients. Upon a remote call, the caller is suspended, the remote procedure is executed, and its results are returned to the caller, which then resumes its execution. A detailed critique of the RPC paradigm is presented in [10]. It is natural to ask whether the RPC mechanism is su cient to solve the micromanagement problem. Let us examine this question via the example 1. Suppose that the entire management program shown there is encoded, as part of the agent code, in terms of a particular action { say, link.handler. A manager will need to execute a remote procedure call invoking execution of the script by the agent. Does this solution o er an adequate model for management? There are two fundamental problems associated with the solution above. The rst is rooted in the synchronous invocation semantics of RPC mechanisms. A management script will typically involve actions to be triggered by independent event occurrences in the agent, asynchronously with the manager's invocation of the RPC. Clearly, the manager's own execution should not be tied with the execution of the management script. An RPC mechanism would block the manager until the completion of the manager script. A second limitation of this model for supporting manager/agent interactions is the static nature of the exported procedures. That is, there is an underlying assumption that the programming of management scripts is done by the agent/managed-object designer at agent-design time (just as server calls are part of server design), rather than at management design time as part of incorporating the agent/managed-objects in a network. 22 Designers of the agent (server) must thus predict and code as agent procedures the entire range of composite management scripts in which this agent may be usefully involved. It is typically not possible to predict at agent-design time all the possible management scenarios in which it may be involved, nor is this desirable, as it would limit the manageability a orded by the product. Furthermore, coding such scripts into the agent would signi cantly increase its complexity and costs. Remote evaluation (REV) [11], goes a step further in permitting the dynamic transfer of programs to remote execution. It allows a procedure (written in LISP) to be transfered from a client to a server where an interpreter will execute it. It thus overcomes the problem of xing all the server procedures that can be invoked remotely at the design time of the server. However, the execution of the remotely evaluated procedure is still synchronous with the execution of the caller (manager): i.e., the REV procedure is executed upon its receipt by the server (agent). Thus, this technique, is not adequate since it does not meet the requirements for management in a distributed environment. In summary, the RPC model fails to provide an adequate solution of the micromanagement problem, as agent designers must predict and code all management scripts required as part of the agent. The remote evaluation model goes a step further in permitting the dynamic transfer of programs to remote execution. However, both models involve synchronous procedure-call interactions whereby the agent executes the management program upon invocation, while the caller is blocked until completion. Thus neither technique proves to be adequate for management in a distributed environment. 8 Non-determinism of management actions Consider an agent to whom a number of management scripts have been delegated (by one or more managers) sharing the same "triggering" event E. When E occurs, some management actions must be executed by the agent. Di erent execution orders may result in substantially di erent results. Consider for instance the following Example 2: Non-determinism of management actions An X.25 controller within a switch includes an agent responsible for managing the controller operations (mostly virtual circuits (VC) operations). The agent has been delegated three management scripts by di erent managers, to be triggered upon the event bu ers-full (i.e., all VC bu ers are full): D1 speci es the following: run local diagnostics to check possible link or controller 23 failures, reboot the controller upon fault; D1 is delegated by a vendor-provided switch manager. D2 speci es: evaluate and report performance parameters, abort some VCs and set limits on local resources usage; D2 is a congestion handler delegated by the network control center. D3 speci es: dump bu ers contents and reset ow-control on VCs; D3 is a owcontrol action pursued by network ow-control protocol (acting in a manager role). This example illustrates typical forms in which management actions distributed through networked systems can lead to unpredictable management actions. First, any order of execution pursued in applying the three actions will lead to very di erent behaviors. Thus, if D1 is executed rst, the controller may be rebooted and then when D2 evaluates the performance parameters it would sample and report to the global manager di erent values than those reported by executing D2 rst and then D1. As a result the controller may pursue unobservable intermittent failures as it reboots and fails again. Global management seeking reporting through D2 could never know about the problem. Second, the roles of managers and agents is e ectively established through the very design of network devices and protocols. A global network manager is not the only entity acting in a role of a manager. Typically, devices would be equipped with local (vendorprovided) software functioning in explicit management roles. Thus, local automated recovery systems, monitoring and diagnostics software also function in management roles. Management roles are often implicitly built-into the very speci cations of certain protocol entities. Thus, ow control procedures of the protocol entity may be triggered by conditions monitored by the protocol entity. In designing an overall management system it is critical that all management actions are considered and that possible interactions among such actions and those pursued by management of a network control center are carefully coordinated. Third, the distribution of management functions among di erent management entities is typically established through ad-hoc evolution. Such distribution of function may contribute to very di cult coordination problems, or even render certain devices and protocol entities unmanageable. Non-determinism can (and should) be avoided by constraining interactions among concurrent management programs. Two management programs are concurrent if they may be simultaneously executed; they are said to be associative if they lead to the same results independent of their execution order. Obviously, the design of a management system 24 must aim to assure that concurrent management programs are associative. The execution orders of non-associative concurrent management programs must be carefully controlled to ensure that the results are deterministic. This bears some similarity to the problem of serializability of transactions in database systems. However, unlike the problem of serializability, where arbitrary serializable execution orders are acceptable, deterministic management requires control over the speci c ordering of concurrent actions. The speci c ordering of non-associative management program executions can re ect management hierarchy (e.g., prioritize programs of higher-level managers) or critical priorities of the program (e.g., execute rst higher priority handlers). 8.1 Interference Management actions may interfere with each other, leading to potential hazards. A management system is coherent when interferences are properly controlled. Interference is typically caused when evaluation of an event or an action causes changes in managed object attributes triggering other events/actions. Coherence may be accomplished by minimizing and controlling interference. For example, interference among event detection may be avoided by preventing side-e ects of detection. It is, however, undesirable to prevent interference among actions and events in general. Consider the following Example 3: An X.25 virtual-circuit manager On (heavy.load) Begin For all active VCs If (VC.thruput > 2.4 Kbs) then VC.thruput := 2.4 Kbs ::End In the example, the event heavy.load triggers changes of the thruput class (set to 2.4Kbs) for some VCs. A user of a VC (e.g., TP4 protocol) may require noti cation of such changes to ensure proper handling. In the absence of TP4 handling of these thruput class changes, the network may crash as a result of the changes. Interference is thus essential to assure communication of changes among managers. Interference is also important in supporting other forms of cooperative management. For example, in a hierarchical management organization, a high-level manager may capture and prevent attempts of lower-level managers to cause certain actions. 25 However, interference may lead to hazardous behaviors. For example, management actions may form a loop, each triggering evaluation of the next action in the loop. This would lead agents to a management livelock. It is thus necessary for managers to control possible interference among management programs and ascertain management coherence. A management system should support coordination primitives for interference control. 9 Status The MAD prototype [12], has successfully demonstrated the delegation model of network management. The prototype includes the following components: 1. A Process Manager which supports all the operations over lightweight DMPIs. 2. An implementation of the delegation protocol, using ASN.1 presentation services and TCP/IP transport services. 3. A Repository providing storage and retrieval facility for DMPs. 4. A Translator supporting DMPs written in C and C++. 5. An SNMP based OCP which allows the Mad Agent to also serve as a SNMP agent. 6. a message-passing IPC service over shared memory, 7. A simpleMad Agent Controller, which implements two preemptive execution schemes, wound-die and wound-wait. 8. Two user interfaces for simple managers: one runs as an window based application over X, and the other, for simpler terminals, based on cursors. 9. Several simple applications to run on Mad as DMPs. The prototype agent can receive, compile, link, and execute nontrivial management scripts from remote manager processes. The scripts execute asynchronously from their delegating manager and take independent action, eliminating the micro-management problem. Managers use the protocol to control the execution of their DMPIs. Several sample DMPs have been written and tested over the prototype. In particular, we implemented a collection of DMPs for the health example described earlier. The Mad agent can optionally incorporate a SNMP agent which mantains a private MIB as an OCP which can be queried directly by SNMP clients. 26 10 ConclusionsCurrent management models, as pursued by standardization e orts, centralize all man-agement functions within managers with agents playing minor roles. This centralizationinduces managers to micro-manage agents, resulting in ine cient and costly managementsystems. It also results in intrinsically unreliable network management, since managerprocesses are turned into sensitive communications and processing bottlenecks whose losscan result in paralysis. Centralized management does not meet the management needsof emerging high-speed, increasingly complex networks and distributed systems. Noris it re ective of emerging cost/performance trade-o s, where processing resources havebecome cheaper, as the costs of mismanagement continue to increase. Network devicesand their agents may require and support as powerful processing capabilities as managerapplications.The traditional Client/Server paradigm does not provide an adequate solution for themicro-management problem, as agent designers must predict and code all managementscripts required as part of the agent. The remote evaluation model goes a step furtherin permitting the dynamic transfer of programs to remote execution. However, bothmodels involve synchronous procedure-call interactions whereby the agent executes themanagement program upon invocation, while the caller is blocked until completion. Thusneither technique proves to be adequate for management in a distributed environment.E ective management requires support for exible, e cient distribution of manage-ment functions and allocation of resources. The Manager-Agent Delegation (MAD) modelproposed in this paper, supports such a distribution. It provides a software frameworkthat enhances the expressive power of management interactions. A manager can delegatean entire management procedure to be executed in the agent's environment, while main-tining control over its execution. Delegated management programs can then be executedlocally at the agent's host environment, without involving the manager unnecessarily.The capabilities and limitations of agents may be re ected in the management scriptinglanguage that they use.From a performance perspective, delegation presents an attractive trade-o . It allowslower communications costs (of management messages) by dynamically allocating man-agement function closer to the real devices. Of course, exibility and expressiveness donot come for free: they require more computing cycles and memory in the MAD agentenvironment. These costs are divided into two parts: a xed component (the built-inMad support) and a varying component (the OCPs and DMPs). There are two reasons27 to justify this cost. The rst one is the increasing costs associated with mismanagementof networks. Such costs will continue to increase, as long as we continue to increase ourreliance on these distributed systems and networks as vital information backbones. Onthe other hand, there is a continous decrease in the cost of computing devices, and inparticular, microprocessors. Thus, a signi cant increase in in the abundance of in pro-cessing cycles in distributed environments is expected. Thus, we advocate increasing thedegree of manageability at the cost of allocating { intelligently { more resources over thenetwork.ACKNOWLEDGMENTS:Most of the current prototype of Mad was implemented by graduate students in theDCC Laboratory of the Computer Science Department of Columbia University. Theyinclude Antonis Maragkos, James Tuller, Chris Wood, Sharon Barkai, Cristina Auerro-coechea and Ben Ohtsu.References[1] Uyless Black. Network Management Standards The OSI, SNMP and CMOL Proto-cols. McGraw Hill, 1992.[2] Branislav N. Meandzija and Jil Westcott, editors. The First IFIP International Sym-posium on Integrated Network Management, Boston, MA, May 1989. North Holland.[3] Iyengar Krishnan and Wolfgang Zimmer, editors. The Second IFIP InternationalSymposium on Integrated Network Management, Washington, DC, April 1991. NorthHolland.[4] Aaron Kershenbaum, Manu Malek, and Mark Wall, editors. Network Managementand Control Workshop. Plenum Press, September 1989.[5] Marshall T. Rose. The Simple Book, An introduction to Management of TCP/IP-based Internets. Prentice Hall, 1991.[6] International Standards Organization ISO. 9596 Information Technology, OpenSystems Interconnection, Common Management Information Protocol Speci cation,May 1990.28 [7] Je rey D. Case, Mark S. Fedor, Martin L. Scho stall, and James R. Davin. A SimpleNetwork Management Protocol (SNMP). RFC 1157, May 1990. DDN NetworkInformation Center, SRI International.[8] Frederick E. Boland, editor. Working Implementation Agreements for Open SystemsInterconnection Protocols. IEEE Computer Society Press, 1990. Vol. 2, Number 2.[9] Andrew D. Birrell and Bruce J. Nelson. Implementing remote procedure calls. ACMTransactions on Computer Systems, 2(1), February 1984.[10] Andrew S. Tanembaum and Robert van Renesse. A critique of the remote procedurecall paradigm. Research into Networks and Distributed Applications, pages 775{783,April 1988.[11] James W. Stamos and David K. Gi ord. Remote evaluation. ACM Transactions onProgramming Languages and Systems, 12(4):537{565, October 1990.[12] German Goldszmidt and Yechiam Yemini. The Design of a Management DelegationEngine. In Proceedings of the IFIP/IEEE International Workshop on DistributedSystems: Operations and Management, Santa Barbara, CA, October 1991.29